Flow-guide synthesis provides a common framework for frame interpolation, where optical flow is typically estimated by a pyramid network, and then leveraged to guide a synthesis network to generate intermediate frames between input frames. In this paper, we present UPR-Net, a novel Unified Pyramid Recurrent Network for frame interpolation. Cast in a flexible pyramid framework, UPR-Net exploits lightweight recurrent modules for both bi-directional flow estimation and intermediate frame synthesis. At each pyramid level, it leverages estimated bi-directional flow to generate forward-warped representations for frame synthesis; across pyramid levels, it enables iterative refinement for both optical flow and intermediate frame. In particular, we show that our iterative synthesis can significantly improve the robustness of frame interpolation on large motion cases. Despite being extremely lightweight (1.7M parameters), UPR-Net achieves excellent performance on a large range of benchmarks. Code will be available soon.

我们为基于运动的视频框架插值提供了一种新颖的简单而有效的算法。现有的基于运动的插值方法通常依赖于预先训练的光流模型或基于U-NET的金字塔网络进行运动估计，该运动估计要么具有较大的模型大小或有限的处理复合物和大型运动案例的容量。在这项工作中，通过仔细整合了中间方向的前射击，轻质特征编码器和相关量为金字塔复发框架，我们得出一个紧凑的模型，以同时估计输入帧之间的双向运动。它的尺寸比PWC-NET小15倍，但可以更可靠，更灵活地处理具有挑战性的运动案例。基于估计的双向运动，我们向前射击输入帧及其上下文特征到中间帧，并采用合成网络来估算扭曲表示的中间帧。我们的方法在广泛的视频框架插值基准测试中实现了出色的性能。代码将很快可用。

聚合邻居功能对于点云分类至关重要。在现有的工作中，不可避免地会选择云中的每个点作为多个聚合中心的邻居，因为所有中心将独立地从整个点云中收集邻居功能。因此，每个点必须反复参与计算，并在内存中生成冗余重复项，从而导致密集的计算成本和记忆消耗。同时，为了追求更高的准确性，以前的方法通常依靠复杂的局部聚合器来提取精细的几何表示，这进一步减慢了分类管道。为了解决这些问题，我们提出了一个新的线性复杂性的本地聚合器，用于点云分类，以应用为应用。具体而言，我们引入一个辅助容器作为锚点，以在源点和聚合中心之间进行交换。每个源点只能将其功能推到一个辅助容器，每个中心点仅从一个辅助容器中拉出特征。这避免了每个源点的重新计算问题。为了促进云点的局部结构的学习，我们使用在线正常估计模块提供可解释的几何信息以增强我们的应用程序建模能力。我们的构建网络比所有以前的基线都更有效，并且在仍然消耗较低的内存的同时，它的空间清晰。合成数据集和真实数据集的实验表明，APP-NET与其他网络相当。它可以每秒处理超过10,000个样本，而单个GPU上的内存少于10GB。我们将在https://github.com/mcg-nju/app-net中发布代码。

将多模式的知识用于抽象性摘要任务是一个正在进行的研究领域，目前的技术遗传了融合，然后代范式。由于计算机视觉和自然语言处理之间的语义差距，当前方法通常将多个数据点视为单独的对象，并依靠注意机制搜索连接以融合在一起。此外，从许多框架中缺少对跨模式匹配的认识会导致性能降低。为了解决这两个缺点，我们提出了一个迭代对比对准框架（ICAF），该框架使用反复对齐和对比度来捕获图像和文本之间的连贯性。具体而言，我们设计了一个经常性比对（RA）层，以逐步研究图像贴片和文本令牌之间的细粒语义关系。在编码过程中的每个步骤中，跨模式对比度损耗被应用以直接优化嵌入式空间。根据Rouge的说法，相关得分和人类评估，我们的模型表现优于MSMO数据集上最新的基线。还进行了有关我们提出的框架和超参数设置的适用性的实验。

Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To that end, we introduce GarDA (Generative Appearance Replay for continual Domain Adaptation), a generative-replay based approach that can adapt a segmentation model sequentially to new domains with unlabeled data. In contrast to single-step unsupervised domain adaptation (UDA), continual adaptation to a sequence of domains enables leveraging and consolidation of information from multiple domains. Unlike previous approaches in incremental UDA, our method does not require access to previously seen data, making it applicable in many practical scenarios. We evaluate GarDA on two datasets with different organs and modalities, where it substantially outperforms existing techniques.

The development of social media user stance detection and bot detection methods rely heavily on large-scale and high-quality benchmarks. However, in addition to low annotation quality, existing benchmarks generally have incomplete user relationships, suppressing graph-based account detection research. To address these issues, we propose a Multi-Relational Graph-Based Twitter Account Detection Benchmark (MGTAB), the first standardized graph-based benchmark for account detection. To our knowledge, MGTAB was built based on the largest original data in the field, with over 1.55 million users and 130 million tweets. MGTAB contains 10,199 expert-annotated users and 7 types of relationships, ensuring high-quality annotation and diversified relations. In MGTAB, we extracted the 20 user property features with the greatest information gain and user tweet features as the user features. In addition, we performed a thorough evaluation of MGTAB and other public datasets. Our experiments found that graph-based approaches are generally more effective than feature-based approaches and perform better when introducing multiple relations. By analyzing experiment results, we identify effective approaches for account detection and provide potential future research directions in this field. Our benchmark and standardized evaluation procedures are freely available at: https://github.com/GraphDetec/MGTAB.

As one of the prevalent methods to achieve automation systems, Imitation Learning (IL) presents a promising performance in a wide range of domains. However, despite the considerable improvement in policy performance, the corresponding research on the explainability of IL models is still limited. Inspired by the recent approaches in explainable artificial intelligence methods, we proposed a model-agnostic explaining framework for IL models called R2RISE. R2RISE aims to explain the overall policy performance with respect to the frames in demonstrations. It iteratively retrains the black-box IL model from the randomized masked demonstrations and uses the conventional evaluation outcome environment returns as the coefficient to build an importance map. We also conducted experiments to investigate three major questions concerning frames' importance equality, the effectiveness of the importance map, and connections between importance maps from different IL models. The result shows that R2RISE successfully distinguishes important frames from the demonstrations.

Compressed videos often exhibit visually annoying artifacts, known as Perceivable Encoding Artifacts (PEAs), which dramatically degrade video visual quality. Subjective and objective measures capable of identifying and quantifying various types of PEAs are critical in improving visual quality. In this paper, we investigate the influence of four spatial PEAs (i.e. blurring, blocking, bleeding, and ringing) and two temporal PEAs (i.e. flickering and floating) on video quality. For spatial artifacts, we propose a visual saliency model with a low computational cost and higher consistency with human visual perception. In terms of temporal artifacts, self-attention based TimeSFormer is improved to detect temporal artifacts. Based on the six types of PEAs, a quality metric called Saliency-Aware Spatio-Temporal Artifacts Measurement (SSTAM) is proposed. Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics. We believe that SSTAM will be beneficial for optimizing video coding techniques.

We propose a distributionally robust return-risk model for Markov decision processes (MDPs) under risk and reward ambiguity. The proposed model optimizes the weighted average of mean and percentile performances, and it covers the distributionally robust MDPs and the distributionally robust chance-constrained MDPs (both under reward ambiguity) as special cases. By considering that the unknown reward distribution lies in a Wasserstein ambiguity set, we derive the tractable reformulation for our model. In particular, we show that that the return-risk model can also account for risk from uncertain transition kernel when one only seeks deterministic policies, and that a distributionally robust MDP under the percentile criterion can be reformulated as its nominal counterpart at an adjusted risk level. A scalable first-order algorithm is designed to solve large-scale problems, and we demonstrate the advantages of our proposed model and algorithm through numerical experiments.

Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving. Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task. To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. The proposed PPGeo is performed in two stages to support effective self-supervised training. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only. As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks. Extensive experiments covering a wide span of challenging scenarios have demonstrated the superiority of our proposed approach, where improvements range from 2% to even over 100% with very limited data. Code and models will be available at https://github.com/OpenDriveLab/PPGeo.